[Core] Improve Large-Scale Resource View Synchronization Through Sync Message Batching - #57641

Awesome writeup and thanks for the contribution @yancanmao. This one will take us a little while to read and digest, but will review shortly :)

@yancanmao I will have more detailed feedback on the implementation once we agree on a design and policy. My only reservation about this change is the requirement to tune the batch size and delay in order to balance responsiveness in the common case w/ performance for large clusters.

One technique that I've seen used in other systems rather than delaying while waiting for a batch is "opportunistic batching" or "dynamic batching". That is, process the sync messages as soon as they come in and as quickly as possible, but as a backlog builds up due to increased load while sending messages out, begin to dynamically batch the messages in the queue.

Let me caveat that I am not exactly sure what the implementation would look like in the "bidi syncer" because I do not deeply understand that code yet. But one strawman implementation would be:

• Instead of the single shared io context that we have today, have two threads: one for receiving sync messages and one for broadcasting them out.
• The receiving thread simply enqueues all received messages into an in-memory buffer (or buffers), similar to what you're currently doing with the sync_message_batch_buffer_.
• The broadcasting thread awaits messages to be added to the in-memory buffer. When at least one is available, it ingests the full buffer, batches as needed, and sends them to the correct destinations.

This might offer a hybrid approach that maintains responsiveness under light load while optimizing for throughput under heavier load. Let me know if this suggestion makes sense. Like I said, I very well might be missing something having not looked as closely at the code as you :)

Another similar approach might be to adjust the batch timeout dynamically based on some form of load metric (e.g., the number of nodes in the cluster or the number of sync messages in the backlog).

@yancanmao I will have more detailed feedback on the implementation once we agree on a design and policy. My only reservation about this change is the requirement to tune the batch size and delay in order to balance responsiveness in the common case w/ performance for large clusters.

One technique that I've seen used in other systems rather than delaying while waiting for a batch is "opportunistic batching" or "dynamic batching". That is, process the sync messages as soon as they come in and as quickly as possible, but as a backlog builds up due to increased load while sending messages out, begin to dynamically batch the messages in the queue.

Let me caveat that I am not exactly sure what the implementation would look like in the "bidi syncer" because I do not deeply understand that code yet. But one strawman implementation would be:

• Instead of the single shared io context that we have today, have two threads: one for receiving sync messages and one for broadcasting them out.
• The receiving thread simply enqueues all received messages into an in-memory buffer (or buffers), similar to what you're currently doing with the sync_message_batch_buffer_.
• The broadcasting thread awaits messages to be added to the in-memory buffer. When at least one is available, it ingests the full buffer, batches as needed, and sends them to the correct destinations.

This might offer a hybrid approach that maintains responsiveness under light load while optimizing for throughput under heavier load. Let me know if this suggestion makes sense. Like I said, I very well might be missing something having not looked as closely at the code as you :)

Another similar approach might be to adjust the batch timeout dynamically based on some form of load metric (e.g., the number of nodes in the cluster or the number of sync messages in the backlog).

Thanks for the insightful comment! The opportunistic buffer idea makes a lot of sense. In fact, RaySyncer and BidiReactor are two asynchronous components, and each BidiReactor already maintains a sending_buffer_ that receives messages from RaySyncer, which can naturally serve as such an opportunistic buffer.

We tested a revised version (the latest commit) that removes the explicit batch size/timeout logic in RaySyncer and instead performs merge-and-flush directly from the reactor’s sending_buffer_ when the sending queue becomes ready.

In our 1250-node PG scheduling test, this version works functionally but shows instability — out of 10 runs, about half finish within ~100 s, while the others take around ~300 s. The variance mainly comes from the random arrival of messages, which affects when the backlog builds up and triggers merge. I also instrumented the batch size and found that it averages around 7 messages per batch, roughly 8× smaller than the manually tuned buffered version.

Overall, the opportunistic buffer approach requires fewer code changes and eliminates manual tuning or timer overhead, but the batching behavior is less predictable and currently less efficient under large-scale, bursty workloads.

As a next step, I think dynamic batching could be a good compromise to remove configuration burden while maintaining more stable performance. For example, we could start with a small timeout (e.g., 100 ms) and adaptively increase or decrease it within a range (0 – 500 ms) based on the observed message arrival rate. This would preserve responsiveness under light load while still batching effectively during spikes, though we’ll need a clear policy for handling sudden sync surges. I’d be happy to hear any input you might have on the policy design.

Thanks for running that experiment @yancanmao! Given the results, my preference would be to start very simple with your original proposal, where the batch size and the batch delay are both configuration parameters. If the batch delay is set to zero it should be equivalent to today's behavior. Then we can empirically decide on a default, and in the future extend the policy to make things dynamic once we understand the performance implications more deeply.

My only lingering concern is that delaying the resource view sync could lead to higher scheduling delays for workloads with high task throughput. Previously, as soon as a task finished and resources became available, the update would be broadcasted and other Raylets would immediately react and schedule a new task to the worker. Now, there can be a short delay before they are notified that resources are available and schedule the task. Do you have any ideas on how to mitigate this? For example, we could skip the batching logic if the cluster is currently at capacity and the update indicates that resources have been freed.

Thanks for running that experiment @yancanmao! Given the results, my preference would be to start very simple with your original proposal, where the batch size and the batch delay are both configuration parameters. If the batch delay is set to zero it should be equivalent to today's behavior. Then we can empirically decide on a default, and in the future extend the policy to make things dynamic once we understand the performance implications more deeply.

My only lingering concern is that delaying the resource view sync could lead to higher scheduling delays for workloads with high task throughput. Previously, as soon as a task finished and resources became available, the update would be broadcasted and other Raylets would immediately react and schedule a new task to the worker. Now, there can be a short delay before they are notified that resources are available and schedule the task. Do you have any ideas on how to mitigate this? For example, we could skip the batching logic if the cluster is currently at capacity and the update indicates that resources have been freed.

Thanks for the detailed follow-up! It makes sense, and we can start simple with configurable batch size and delay, where setting delay=0 preserves the current behavior. In the meantime, I agree that we can revisit dynamic batching later once we gather more empirical data from our production deployment.

For your concern about scheduling delays, that’s a good point. I think RaySyncer is a relatively isolated component in the Ray cluster and doesn’t need to be directly aware of the cluster workload. Instead, it should focus on its own performance metrics — primarily queue length.

Since batching is primarily designed for large-scale deployments, a small millisecond-level delay should be acceptable as long as the overall throughput remains high. In practice, during high-frequency task scheduling, resource view updates occur very frequently, generating many sync messages that can overload the GCS syncer. In such cases, batching can actually reduce end-to-end latency by mitigating broadcast overload rather than increasing it.

So in my view, we can disable batching for small clusters to preserve responsiveness, but keep it enabled for large-scale or high-frequency scheduling scenarios. We also think it makes sense to leverage the existing sending_buffer_ as an optimistic buffer — it’s straightforward and already fits well with the current async design. We’ll continue evaluating this approach in production to collect more metrics and understand its stability under different workloads.

@yancanmao ok, so to summarize the plan forward:

• By default we will implement only opportunistic batching (I see no downside to this), but no batching delay.
• We will make the batching delay and the max batch size configurable parameters.
• Over time, we can evolve the policy and automatically tune the batching parameters based on empirical data such as cluster size.

How does that sound?